That is right, a habitual residence, having a house or apartment in Portugal does not make you a habitual residence and subject to Portuguese tax laws.
From the EU web-site whose rules govern habitual residence because being a habitual residence means you can also claim benefits not just in Portugal but in the EU.
The term “habitual resident” means that you must have your “centre of interests” in Portugal. It also refers to permanence: a person who has been in Portugal for some time and intends to stay here for the foreseeable future. Using it as a vacation destination for a period of time each year and have all of your income, mail, bills as well as your family and long-time friends that you spend time with more than 183 days a year in the US, means you center of interest is in the US. Therefore you are not a habitual resident. Deloitte went through the entire scenario and spending up to almost 6 months a year in Portual with all of your income, benefits, affairs, family, long-time friends in the US means you do not need to pay taxes in Portugal on US derived income. On Portuguese derived income you must pay tax and report it to the US and may have to also pay tax on. The tax authority gave the opinion in my case, so I know what I am going to experience.