That depends on which canton you reside in. Swiss federal law gives the general public the right to access information held by government agencies. However, where tax information is concerned, cantonal governments are free to exercise taxpayer protection laws at their discretion.
In the Canton of Zurich, anybody can request access to information held by the tax office. The same holds true in the cantons of Vaud and Neuchâtel. Information provided includes your personal information and references to your taxable income and wealth.
Other cantons only allow access to tax information when certain conditions are met. Typically, those applying for access to information must either prove a direct economic interest in the information (example: the person in question owes them money) or provide written permission from the relevant taxpayer. Cantons with these limitations include Bern, Fribourg, Lucerne, St. Gallen, Schaffhausen, Appenzell-Ausserrhoden, Valais and Uri. All other cantons do not generally allow third parties to access sensitive tax information.
In some cantons, such as Fribourg, only the tax information of private individuals can be accessed while other entities (businesses, for example) benefit from tax secrecy. In Valais, third parties can only request access to tax information held by the tax office of their municipality of residence. Fribourg only extends the right to access tax information to cantonal residents.
Tax offices which provide tax information to third parties generally charge a fee for the service.
Allowing unknown individuals to access your sensitive tax information can potentially put you at risk of numerous kinds of abuse. For example, making information about your taxable income or personal wealth available to the general public can result in your being targeted by financial predators.
Considering the disadvantages, it is worth noting that in many cantons you can have your sensitive personal information protected from prying eyes.
In Zurich, for example, you can request that your information be blocked from the general public. This applies to all public offices and prevents members of the public from accessing your information unless they can prove that the blockage prevents them from exercising their civil rights.
In order to do this, you need to submit a form requesting that your sensitive private information be blocked to outside requests. Your data will still be available to government officials and tax authorities, but it won't be available to the general public. In addition to protecting sensitive information held by the tax office, you can also request that your data at the municipal registry office and the traffic department be withheld from the public.
If you reside in a canton which allows public access to tax information and you do not specifically apply to have your data withheld, your information can, by default, be shared with anyone who ask for it. The onus rests on you to decide whether or not you want to withhold your tax data from the public and to take action if necessary.